{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {},
  "cells": [
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "   <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/master/examples/model_experiments/model_diagnostics/custom_metrics_basics.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
        "</td>\n",
        "\n",
        "<td>\n",
        "<a href=\"https://github.com/Labelbox/labelbox-python/tree/master/examples/model_experiments/model_diagnostics/custom_metrics_basics.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "----\n",
        "\n",
        "# Model Diagnostics - Custom Metrics Basics\n",
        "\n",
        "\n",
        "* Measuring model quality is critical to efficiently building models. It is important that the metrics used to measure model quality closely align with the business objectives for the model. Otherwise, slight changes in model quality, as they related to these core objectives, are lost to noise. Custom metrics enables users to measure model quality in terms of their exact business goals. By incorporating custom metrics into workflows, users can:\n",
        "    * Iterate faster\n",
        "    * Measure and report on model quality\n",
        "    * Understand marginal value of additional labels and modeling efforts\n",
        "\n",
        "\n",
        "* For an end-to-end demo of diagnostics using custom metrics checkout this [notebook](custom_metrics_demo.ipynb)\n",
        "\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "## Environment Setup\n",
        "\n",
        "Install dependencies"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "!pip install -q \"labelbox[data]\""
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "Import libraries"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "import labelbox.types as lb_types\n",
        "import labelbox as lb\n",
        "import uuid\n",
        "import json"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Custom Metrics\n",
        "* Users can provide metrics at the following levels of granularity:\n",
        "    1. data rows\n",
        "    2. features\n",
        "    3. subclasses\n",
        "* Additionally, metrics can be given custom names to best describe what they are measuring.\n",
        "    \n",
        "* Limits and Behavior:\n",
        "    * At a data row cannot have more than 20 metrics\n",
        "    * Metrics are upserted, so if a metric already exists, its value will be replaced\n",
        "    * Metrics can have values in the range [0,100000]\n",
        "* Currently `ScalarMetric`s and `ConfusionMatrixMetric`s are supported. "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "### ScalarMetric\n",
        "    * A `ScalarMetric` is a metric with just a single scalar value."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "from labelbox.data.annotation_types import (ScalarMetric,\n",
        "                                            ScalarMetricAggregation,\n",
        "                                            ConfusionMatrixMetric)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "data_row_metric = ScalarMetric(metric_name=\"iou_custom\", value=0.5)\n",
        "\n",
        "feature_metric = ScalarMetric(metric_name=\"iou_custom\", feature_name=\"cat\", value=0.5)\n",
        "\n",
        "subclass_metric = ScalarMetric(metric_name=\"iou_custom\",\n",
        "                               feature_name=\"cat\",\n",
        "                               subclass_name=\"organge\",\n",
        "                               value=0.5)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### ConfusionMatrixMetric\n",
        "- A `ConfusionMatrixMetric` contains 4 numbers [True postivie, False Postive, True Negative, False Negateive]\n",
        "- Confidence is also supported a key value pairs, where the score is the key and the value is the metric value.\n",
        "- In the user interface, these metrics are used to derive precision,recall, and f1 scores. The reason these are not directly uploaded is that the raw data allows us to do processing on the front end.\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "data_row_metric = ConfusionMatrixMetric(metric_name=\"50pct_iou\",\n",
        "                                        feature_name=\"cat\",\n",
        "                                        subclass_name=\"organge\",\n",
        "                                        value=[1, 0, 1, 0])\n",
        "\n",
        "feature_metric = ConfusionMatrixMetric(metric_name=\"50pct_iou\",\n",
        "                                       feature_name=\"cat\",\n",
        "                                       subclass_name=\"organge\",\n",
        "                                       value=[1, 0, 1, 0])\n",
        "\n",
        "subclass_metric = ConfusionMatrixMetric(metric_name=\"50pct_iou\",\n",
        "                                        feature_name=\"cat\",\n",
        "                                        subclass_name=\"organge\",\n",
        "                                        value=[1, 0, 1, 0])"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Confidence\n",
        "* Users can provide confidence scores along with metrics\n",
        "* This enables them to explore their model performance without necessarily knowing the optimal thresholds for each class.\n",
        "* Users can filter on confidence and value in the UI to perform powerful queries.\n",
        "* The keys represent a confidence score (must be between 0 and 1) and the values represent either a scalar metric or for confusion matrix metrics [TP,FP,TN,FN]"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "confusion_matrix_metric_with_confidence = ConfusionMatrixMetric(\n",
        "    metric_name=\"confusion_matrix_50pct_iou\",\n",
        "    feature_name=\"cat\",\n",
        "    subclass_name=\"organge\",\n",
        "    value={\n",
        "        0.1: [1, 0, 1, 0],\n",
        "        0.3: [1, 0, 1, 0],\n",
        "        0.5: [1, 0, 1, 0],\n",
        "        0.7: [1, 0, 1, 0],\n",
        "        0.9: [1, 0, 1, 0]\n",
        "    })\n",
        "\n",
        "scalar_metric_with_confidence = ScalarMetric(metric_name=\"iou_custom\",\n",
        "                                             value={\n",
        "                                                 0.1: 0.2,\n",
        "                                                 0.3: 0.25,\n",
        "                                                 0.5: 0.3,\n",
        "                                                 0.7: 0.4,\n",
        "                                                 0.9: 0.3\n",
        "                                             })"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Aggregations\n",
        "* This is an optional field on the `ScalarMetric` object (by default it uses Arithmetic Mean).\n",
        "* Aggregations occur in two cases:\n",
        "    1. When a user provides a feature or subclass level metric, Labelbox automatically aggregates all metrics with the same parent to create a value for that parent.\n",
        "        * E.g. A user provides cat and dog iou. The data row level metric for iou is the average of both of those.\n",
        "        * The exception to this is when the data row level iou is explicitly set, then the aggregation will not take effect (on a per data row basis). \n",
        "    2. When users create slices or want aggregate statistics on their models, the selected aggregation is applied."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "\"\"\"\n",
        "If the following metrics are uploaded then\n",
        "in the web app, users will see:\n",
        "true positives dog = 4\n",
        "true positives cat = 3\n",
        "true positives = 7\n",
        "\"\"\"\n",
        "\n",
        "feature_metric = ScalarMetric(metric_name=\"true_positives\",\n",
        "                              feature_name=\"cat\",\n",
        "                              value=3,\n",
        "                              aggregation=ScalarMetricAggregation.SUM)\n",
        "\n",
        "feature_metric = ScalarMetric(metric_name=\"true_positives\",\n",
        "                              feature_name=\"dog\",\n",
        "                              value=4,\n",
        "                              aggregation=ScalarMetricAggregation.SUM)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Built-in Metrics:\n",
        "* The SDK Provides a set of default metrics that make metrics easy to use.\n",
        "1. `confusion_matrix_metric()`\n",
        "    * Computes a single confusion matrix metric for all the predictions and labels provided. \n",
        "2. `miou_metric()`\n",
        "    * Computes a single iou score for all predictions and labels provided    \n",
        "3. `feature_confusion_matrix_metric()`\n",
        "    * Computes the iou score for each of the classes found in the predictions and labels\n",
        "4. `feature_miou_metric()`\n",
        "     * Computes a confusion matrix metric for each of the classes found in the predictions and labels\n",
        "------\n",
        "* Note that all of these functions expect the prediction and ground truth annotations to correspond to the same data row. These functions should be called for each data row that you need metrics for."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "from labelbox.data.metrics import (feature_miou_metric, miou_metric,\n",
        "                                   confusion_matrix_metric,\n",
        "                                   feature_confusion_matrix_metric)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "predictions = [\n",
        "    lb_types.ObjectAnnotation(name=\"cat\",\n",
        "                     value=lb_types.Rectangle(start=lb_types.Point(x=0, y=0),\n",
        "                                     end=lb_types.Point(x=10, y=10)))\n",
        "]\n",
        "\n",
        "ground_truths = [\n",
        "    lb_types.ObjectAnnotation(name=\"cat\",\n",
        "                     value=lb_types.Rectangle(start=lb_types.Point(x=0, y=0), end=lb_types.Point(x=8,\n",
        "                                                                      y=8)))\n",
        "]"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "print(feature_miou_metric(ground_truths, predictions))\n",
        "print(miou_metric(ground_truths, predictions))\n",
        "print(confusion_matrix_metric(ground_truths, predictions))\n",
        "print(feature_confusion_matrix_metric(ground_truths, predictions))"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "# Adjust iou for iou calcuations.\n",
        "# Set it higher than 0.64 and we get a false postive and a false negative for the other ground truth object.\n",
        "print(feature_confusion_matrix_metric(ground_truths, predictions, iou=0.9))"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "\n",
        "# subclasses are included by default\n",
        "predictions = [\n",
        "    lb_types.ObjectAnnotation(\n",
        "        name=\"cat\",\n",
        "        value=lb_types.Rectangle(start=lb_types.Point(x=0, y=0), end=lb_types.Point(x=10, y=10)),\n",
        "        classifications=[\n",
        "            lb_types.ClassificationAnnotation(\n",
        "                name=\"height\",\n",
        "                value=lb_types.Radio(answer=lb_types.ClassificationAnswer(name=\"tall\")))\n",
        "        ])\n",
        "]\n",
        "\n",
        "ground_truths = [\n",
        "    lb_types.ObjectAnnotation(\n",
        "        name=\"cat\",\n",
        "        value=lb_types.Rectangle(start=lb_types.Point(x=0, y=0), end=lb_types.Point(x=10, y=10)),\n",
        "        classifications=[\n",
        "            lb_types.ClassificationAnnotation(\n",
        "                name=\"height\",\n",
        "                value=lb_types.Radio(answer=lb_types.ClassificationAnswer(name=\"short\")))\n",
        "        ])\n",
        "]\n",
        "conf_matrix_metrics = feature_confusion_matrix_metric(ground_truths,\n",
        "                                                      predictions)\n",
        "iou_metrics = feature_confusion_matrix_metric(ground_truths,\n",
        "                                              predictions,\n",
        "                                              include_subclasses=False)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "print(\"Subclasses:\", conf_matrix_metrics[0].value)\n",
        "print(\"Excluding Subclasses:\", iou_metrics[0].value)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Uploading Custom Metrics\n",
        "* Custom metrics are uploaded the same way as any MEA upload. NDJson must be created. Fortunately this is made easy with converter functions.\n",
        "* First construct a metric annotation in one of two ways:\n",
        "    1. Manually\n",
        "    2. Using one of the provided functions `feature_miou_metric`, `miou_metric`, `confusion_matrix_metric`, `feature_confusion_matrix_metric`.\n",
        "* Then add the metric annotation to a label ( This step associates the metrics with a data row)\n",
        "* Convert to ndjson and upload"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Continuing with the last example:\n",
        "global_key = \"<global_key>\"\n",
        "metrics = [*conf_matrix_metrics, *iou_metrics]\n",
        "labels = [\n",
        "    lb_types.Label(data=lb_types.ImageData(global_key=global_key), annotations=metrics)\n",
        "]\n",
        "# We can upload these metric with other annotations\n",
        "#model_run.add_predictions(f'diagnostics-import-{uuid.uuid4()}', labels)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    }
  ]
}